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A  wall  of  blue  curtains  extends  along  the  entrance  to  our 
machine  room,  camouflaging  the  presence  of  a  portion 
of  the  support  infrastructure  for  the  High  Performance 
Computing  (HPC)  systems  located  there.  We’d  like  to 
pull  back  the  blue  curtain  of  the  NAVO  MSRC  and  reveal 
the  team  that  works  to  keep  our  center  one  of  the  top 
supercomputing  centers  in  the  world. 

As  some  user  services  are  consolidated  across  the  High 
Performance  Computing  Modernization  Program  (HPCMP), 
our  center  is  redefining  its  role  in  supporting  the  Program 
and  its  user  community.  The  center  is  always  growing, 
changing,  and  working  diligently  to  meet  the  needs  of  the 
users,  the  program,  and  the  Department  of  Defense.  We 
continue  to  provide  Tier  2  and  Tier  3  support  to  our  users. 
Our  center  staff  strives  to  attain  the  height  of  technical 
knowledge  and  expertise  across  numerous  disciplines  in 
order  to  develop  and  maintain  a  dependable,  leading-edge 
high  performance  computing  infrastructure  for  our  users. 

In  response  to  user  requests  and  suggestions,  we  have 
revamped  our  website  to  provide  users  a  more  thoughtfully 
organized  interface  to  the  center.  And  in  response  to  the 
HPCMP’s  recognition  of  a  need  for  real-time  computing, 
a  team  of  developers  has  been  working  to  create  a  secure, 
web-based  system  for  requesting  dedicated  time  on  a 
portion  of  our  P4+  system,  KRAKEN.  This  reservation 
request  system  will  be  deployed  in  the  fall  of  2007. 

The  NAVO  MSRC  is  co-lead  on  the  new  HPCMP-wide  Mass 
Storage  Initiative,  and  we  are  using  our  current  experience 
in  providing  a  combined  7.8  petabytes  in  both  archival 
and  disaster  recovery  storage  to  craft  an  innovative  and 
forward-thinking  solution  for  this  ever-growing  problem. 

The  center  also  provides  HPC  support  and  resources  to 
the  Navy’s  Meteorology  and  Oceanography  (METOC) 
community.  This  METOC  community  relies  on  the  NAVO 
MSRC’s  well-established  HPC  system  stability  and  uptime 


to  develop  climate,  weather,  and  ocean  forecast  products 
that  are  pushed  to  a  variety  of  customers  around  the  clock. 
In  addition  to  a  number  of  talented  HPC,  storage,  and 
queuing  system  administrators,  we  also  have  a  contingent 
of  operator  staff  in  place  24  hours  a  day,  7  days  a  week 
to  provide  constant  monitoring  of  center  systems  and  to 
ensure  that  after-hours  problems  are  handled  swiftly. 

A  small  but  dedicated  NAVO  MSRC  team  has  worked  for 
several  months  to  organize  and  produce  the  largest  exhibit 
presence  the  HPCMP  has  ever  had  on  the  Supercomputing 

The  Team  Behind 
the  Curtain 

Christine  Cuicchi 

Computational  Science  and  Applications  Lead, 
NAVO  MSRC 


Conference  (SC07)  show  floor — bringing  together  Major 
Shared  Resource  Centers,  Allocated  Distributed  Centers, 
and  HPCMP  initiatives  together  in  an  information-rich 
environment  in  a  40  x  50  foot  floor  space  in  the  Research 
Exhibit  area. 

In  short,  our  team  is  dedicated  to  meeting  and  exceeding 
the  needs  of  the  HPCMP  We  hope  you’ve  enjoyed 
the  glance  behind  the  curtain,  and  we  look  forward  to 
continuing  to  provide  excellent  HPC  services  and  support 
to  the  DoD  HPCMP 
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performance  computing  (HPC)  resources,  including 
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UISASTER  RECOVERY  EFFORT 

nwurjnif 


-Providing  specialized 

24x7  SUPPORT  TO  THE 

Naval  Meteorology  and 
Oceanography  (MBTOC) 

OPERATIONAL  FORECASTINC 
COMMUNITY 


-Providing  over  58  teraflops  of  computing  power 
TO  THE  DoD  HPCMP  USER  COMMUNITY 

IBM  P5+  -  Babbage  -  22  TFLOPs/3072  processors 
IBM  P5+  -  Pascal  -  u,  Tflops  /1920  processors 
IBM  P4+  -  Kraken  -  19  TFL0PS/29M  processors 
IBM  Pi,+  -  Romulus  -  3  Tflops/512  processors 


Design  of  Energetic  Ionic  Liquids 

Jerry  A.  Boatz,  Air  Force  Research  Laboratory,  Space  and  Missile  Propulsion  Division,  Edwards  AFB,  CA 
Hui  Li,  Department  of  Chemistry,  University  of  Nebraska-Lincoln 
Mark  S.  Gordon,  Department  of  Chemistry,  Iowa  State  University 


An  essential  need  of  the  U.S.  Air 
Force  is  the  discovery,  development, 
and  fielding  of  new,  energetic 
materials  for  advanced  chemical 
propulsion  in  rocket  and  missile 
applications.  Some  of  the  key  factors 
driving  the  requirement  for  new 
chemical  propellants  include  (a) 
improved  performance  in  terms  of 
increased  specific  impulse  and  density, 
(b)  reduced  sensitivity  to  external 
stimuli  such  as  impact,  friction, 
shock,  and  electrostatic  discharge, 
and  (c)  mitigation  of  environmental 
and  toxicological  hazards  (and 
the  resulting  costs)  associated  with 
currently  used  propellants. 

A  class  of  compounds  that  can 
potentially  meet  these  requirements  is 
known  as  ionic  Liquids  (ILs),  which 
are  chemical  salts  with  unusually 
low  melting  points.  The  physical  and 
chemical  properties  of  ILs  render 


them  useful  for  many  purposes,  most 
notably  as  environmentally  benign 
(“green”)  solvents/reaction  media  but 
also  as  catalysts,  electrolytes,  etc.1 
From  a  Department  of  Defense  (DoD) 
perspective,  ILs  are  being  explored 
as  new  propellants,  explosives,  and 
munitions. 2 

The  Air  Force,  in  particular,  is 
interested  in  ILs  as  potential 
replacements  for  currently  used 
monopropellants  such  as  hydrazine, 
which  is  carcinogenic,  highly 
toxic,  and  has  relatively  modest 
performance  characteristics. 

In  contrast,  many  ILs  have  superior 
densities  and  specific  impulses  as  well 
as  significantly  reduced  sensitivity  and 
toxicity  characteristics.  Furthermore, 
their  properties  can  be  carefully  tuned 
via  the  choice  of  the  component  ions. 
The  overall  objective  of  the  Design 
of  Energetic  Ionic  Liquids  challenge 


project  is  to  address  several  key 
technical  issues  and  challenges 
associated  with  the  characterization, 
design,  and  development  of  ILs  as 
new  monopropellants.  Among  these, 
for  example,  are  a  fundamental 
understanding  of  the  (in)stability 
of  ILs,  the  intrinsic  nature  of  the 
short-  and  long-range  structure  and 
interactions  between  the  component 
ions,  2e-f  and  identification  of  the 
key  steps  in  the  initial  stages  of 
decomposition  and  combustion.  2a-c 
The  research  described  in  this  article 
is  focused  on  characterization  of 
the  structures  and  stabilities  of  ion 
pair  clusters  and  prediction  of  their 
interaction  energies  in  the  gas  phase. 
Our  computational  approach  utilizes 
quantum  chemical  methods  for 
prediction  of  ion  pair  structures  and 

Continued  Next  Page... 


Figure  1.  MP2/aug-cc-pVDZ  optimized  structures  of  two  pairs  of  1,2,4-triazolium  (1,2,4-triazole)  and  dinitramide 
(dinitramine)  molecules.  H  is  white,  C  is  gray,  O  is  red,  N  is  blue. 
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interaction  energies.  In  particular, 
geometry  optimizations  were 
performed  using  second-order 
perturbation  theory3  (MP2,  also  known 
as  MBPT2)  with  the  aug-cc-pvdz  basis 
set,4  denoted  as  MP2/aug-cc-pvdz. 

Relative  energies  were  refined  using 
a  systematic  series  of  single-point 
energy  calculations  at  the  MP2  and 
coupled  cluster  (e.g.,  singles  and 
doubles  with  a  perturbative  estimate 
of  triples,  CCSD(T)5)  levels  of  theory. 
Specifically,  MP2/cc-pvdz,  MP2/aug- 
cc-pvdz,  and  CCSD(T)/cc-pvdz  energy 
calculations  were  combined  to  obtain 
estimated  CCSD(T)/aug-cc-pvdz 
relative  energies.  All  computations 
were  performed  using  the  GAMESS 
quantum  chemistry  code. 6 
MP2  and  coupled  cluster  (CC) 
calculations  in  GAMESS  utilize  a  library 
of  communications  routines  known 
as  the  Distributed  Data  Interface 
(DDI),7  a  high-level  communications 
layer  operating  between  GAMESS 
and  the  underlying  message-passing 
protocols  (Shared  Memory  (SHMEM), 
Message  Passing  Interface  (MPI), 
Low-level  Application  Programming 
Interface  (LAPI),  or  sockets  within  a 
Transmission  Control  Protocol/Internet 
Protocol  (TCP/IP)  stack.) 


In  the  case  of  the  Naval 
Oceanographic  Office  Major 
Shared  Resource  Center  (NAVO 
MSRC)  IBM  systems  KRAKEN 
and  BABBAGE,  DDI  uses  MPI  for 
intranode  communications  and  the 
LAPI  protocol  for  messages  between 
nodes.  These  types  of  calculations 
have  significant  memory  requirements 
and  therefore  are  well  suited  for 
execution  on  systems  with  large 
amounts  of  memory  per  node,  such 
as  BABBAGE. 

Coupled  cluster  calculations  are 
especially  memory  intensive  and, 
as  implemented  in  GAMESS  with 
DDI,  utilize  a  threefold  hierarchy  of 
memory.  First,  a  modest  amount  of 
Replicated  Data  (RD)  is  exclusively 
assigned  to  each  core.  Similarly,  a 
block  of  Node-specific  Data  (ND)  is 
reserved  on  each  node  and  is  shared 
by  all  the  cores  on  that  node.  The 
remaining  memory  on  each  node  is 
collectively  shared  by  all  cores  as  a 
large,  single  pool  of  Distributed  Data 
(DD).  Therefore,  the  required  Memory 
(Mcc)  per  node  for  CC  calculations 
is  Mcc  =  P*(RD)  +  (ND)  +  (DD)/N, 
where  “P”  and  “N”  are  the  number  of 
cores  per  node  and  the  total  number 
of  nodes,  respectively,  used  in  the 
computation. 


The  values  of  RD,  ND,  and  DD  are 
determined  by  the  specifics  of  the 
calculation,  whereas  suitable  values 
of  P  and  N  are  dictated  by  the 
hardware,  specifically,  the  amount 
of  accessible  physical  memory  per 
node.  If  necessary,  P  can  be  chosen 
to  be  smaller  than  the  number  of 
available  cores  per  node  Pmax 
in  order  to  reduce  the  amount  of 
required  memory  per  node.  Table  1 
summarizes  the  memory  requirements 
for  CCSD(T)  calculations  using  a 
series  of  increasingly  large  basis  sets. 
Only  the  smallest  calculation 
(CCSD(T)/cc-pvdz)  could  be 
performed  within  the  constraints  of 
the  hardware  (Pmax  and  Mmax)  and 
the  challenge  queue  limits  (Nmax 
and  Tmax,  see  Table  2.)  In  principle, 
the  CCSD(T)/6-311++G(d,p)8  and 
CCSD(T)/aug-cc-pvdz  calculations 
could  be  run  on  the  pair  of  “bigmem” 
nodes,  but  the  estimated  required 
wall  time  of  the  former,  on  the  order 
of  100  days,  is  prohibitively  long. 
Conversely,  this  calculation  would 
be  within  the  realm  of  practicality  if 
—  100  bigmem  nodes  were  available. 
One  of  the  specific  ion  combinations 
considered  in  this  work  is  the  1,2,4- 
triazolium  cation  ([C2N3H4] +) 
paired  with  the  dinitramide  anion 
([N(N02)2]-)-  Of  the  numerous 
structures  found  for  the  two  pairs 
of  1,2,4-triazolium  and  dinitramide 
ions,  or  the  pairs  of  corresponding 
neutral  1,2,4-triazole  and  dinitramine 
molecules,  the  most  stable  MP2/aug- 
cc-pvdz  optimized  geometries  are 
shown  in  Figure  1. 

In  the  ionic  structure,  each  1,2,4- 
triazolium  forms  two  hydrogen  bonds, 
via  the  hydrogens  on  the  N  atoms, 
to  the  O  atoms  of  the  dinitramide 
ions.  Interestingly,  this  structure 
exhibits  parallel  stacking  of  the  two 
cationic  1,2,4-triazolium  rings.  The 
interplane  distance  is  —3.2  A,  with  a 
parallel  displacement  of  —1.4  A.  The 
corresponding  neutral  tetramer  shows 
a  similar  parallel  stacking  arrangement 
of  the  triazole  rings. 


Basis  Set 
(#  of  AOs) 

RD 

(MW/core) 

ND 

(MW/node) 

DD  (MW) 

P 

N 

MCC 

(MW/node) 

cc-pvdz 

8 

1,175 

4,950 

16 

64 

1,381 

(376) 

6-31 1  ++G(d,p) 

16 

64 

3,900b 

22 

3,298 

16,000 

1 

64 

3,570b 

(580) 

16 

2 

11,474c 

aug-cc-pvtz 

26 

3,875 

19,150 

1 

64 

4,200b 

(1268) 

1 

2 

13,476c 

1 

64 

21,105b 

aug-cc-pvqz 

330 

18,493 

146,000 

1 

2 

91,823d 

aug-cc-pvqz 

1 

64 

75,475b 

2495 

60,370 

807,000 

(2228) 

1 

2 

466,365d 

a.  “MW”  denotes  megawords.  (106  64-bit  words) 

b.  Exceeds  amount  of  usable  physical  memory  on  each  standard  node.  (See  Table  2) 

c.  Fits  within  usable  physical  memory  on  each  bigmem  node,  but  execution  time  is  prohibitively  long. 

d.  Exceeds  amount  of  usable  physical  memory  on  each  bigmem  node.  (See  Table  2) 

Table  1.  Memory  requirements  for  CCSD(T)  single  point  energy 
calculations. 
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Furthermore,  it  is  of  interest  to 
determine  the  cluster  size  at  which 
the  ion  pair  structures  become  more 
stable  than  the  corresponding  neutral 
pair  structures.  A  previous  study 
predicted  that  ion  pair  dimers  are 
typically  higher  in  energy  than  neutral 
pair  dimers. 2c  Including  zero  point 
vibrational  energy  (ZPVE)  corrections, 
the  ionic  tetramer  in  Figure  1  is  1.2 
kilocalorie/mole  (kcal/mol)  lower  than 
that  of  the  neutral  one. 

The  MP2  method  tends  to  predict 
higher  energies  for  ionic  species  vs. 
neutral  species,  2c  so  more  accurate 
CCSD(T)/aug-cc-pVDZ  energy 
calculations  of  these  two  tetramer 
structures  were  desired.  However, 
since  the  computational  cost  of 
CCSD(T)/aug-cc-pVDZ  is  prohibitive, 
these  energies  were  approximated 
from  the  MP2/aug-cc-pVDZ 


energies  by  estimating  the  electron 
correlation  energy  differences  using 
three  independent  methods:  (1)  the 
differences  between  the  MP2/cc-pVDZ 
and  CCSD(T)/cc-pVDZ  energies  of 
the  tetramers,  (2)  the  differences 
between  the  MP2/aug-cc-pVDZ  and 
CCSD(T)/aug-cc-pVDZ  energies  of 
the  twelve  pairs  of  dimers  in  these 
two  tetramers,  and  (3)  the  differences 
between  the  MP2/aug-cc-pVDZ  and 
CCSD(T)/aug-cc-pVDZ  energies  of 
the  eight  monomers  in  these  two 
tetramers.  Using  these  three  methods, 
and  including  ZPVE  corrections,  the 
estimated  CCSD(T)/aug-cc-pVDZ 
energy  of  the  ionic  tetramer  is  lower 
than  that  of  the  neutral  tetramer 
by  5.7,  7.3,  and  7.7  kcal/mol, 
respectively. 

In  conclusion,  quantum  chemical 
calculations  suggest  that  cation-cation 


parallel  stacking  structures  can  exist 
in  very  small  ionic  clusters  such  as 
two  1,2,4-triazolium  cations  and  two 
dinitramide  anions.  Furthermore, 
for  two  pairs  of  1,2,4-triazolium  and 
dinitramide,  ionic  structures  are  more 
stable  than  the  corresponding  neutral 
structures. 

Finally,  it  should  be  noted  that  lower 
theoretical  methods,  do  not  include 
the  effects  of  electron  correlation, 
such  as  Hartree-Fock,  do  not  predict 
a  parallel  stacking  geometry  of  the 
rings.  Therefore,  it  is  essential  to 
utilize  correlated  methods  such  as 
MP2  and  CCSD(T)  in  order  to  obtain 
proper  descriptions  of  the  structures 
and  interaction  energies  of  these  ion 
clusters.  The  structural  motifs  and 
interaction  patterns  found  in  this  study 
provide  new  understanding  of  ionic 
materials  with  aromatic  rings. 


Queue 

Cores  per  node 
(Nmax) 

Maximum  #  of  nodes 
(Nmax) 

Maximum  memory/node 
Mmax 
(MW/node) 

Maximum  wall  time  Tmax 
(hours) 

Challenge 

16 

64 

-3,500 

48 

Bigmem 

16 

2 

-15,500 

48* 

a.  “MW"  denotes  megawords.  (106  64-bit  words) 

*  48  hour  limit  obtained  via  special  request.  The  default  wall  time  limit  of  the  bigmem  queue  is  12  hours. 


Table  2.  Challenge  and  bigmem  queue  characteristics  on  BABBAGE. 
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Supporting  the  Navy's  METOC  Operational 
Community 

Christine  Cuicchi,  Computational  Science  and  Applications  Lead,  NAVO  MSRC 
Dr.  Frank  Bub,  Ocean  Modeling  Technical  Lead,  Naval  Oceanographic  Office 


The  NAVO  MSRC  serves  a  unique 
function  within  the  HPCMP:  the  center 
supports  the  Navy’s  Meteorology  and 
Oceanography  (METOC)  operational 
forecasting  community  on  a  24  hours 
a  day,  seven  days  a  week  basis. 

The  METOC  operational  forecasting 
group  located  at  the  Naval 
Oceanographic  Office  (NAVOCEANO) 
run  a  number  of  oceanographic 
models  on  the  unclassified  and 
classified  HPC  systems  at  the  center 
in  order  to  provide  oceanographic 
forecast  products  to  the  Navy’s  fleet. 

NAVOCEANO  operational  modelers 
run  a  number  of  models  from  the 
Computational  Technology  Areas 
(CTA)  of  Climate  Weather  Ocean 
(CWO)  and  Computational  Fluid 
Dynamics  (CFD)  on  NAVO  MSRC 
HPC  systems  several  times  a  day. 
These  applications  include  the 


Navy  Coastal  Ocean  Model  gib  Domain 

Currents  at  00100  meters  036  hr  FORECAST  valid  10  OCT  07  1200Z 


Modular  Ocean  Data  Assimilation 
System  (MODAS),  Relocatable 
Navy  Coastal  Ocean  Model  (RELO 
NCOM),  the  Navy  Coupled  Ocean 
Data  Assimilation  System  (NCODA), 
the  Global  Navy  Coastal  Ocean 
Model  (NCOM),  the  Shallow  Water 
Assimilation  Forecast  System 
(SWAFS),  the  Wave  Analysis  Model 
(WAM),  and  the  Simulating  Waves 
Nearshore  model  (SWAN). 

High  performance  computing 
resources  are  required  as  these  models 
are  run  not  only  numerous  areas 
of  interest  throughout  the  world’s 
oceans,  but  also  at  varying  resolutions, 
some  of  which  are  as  high  as  l/50th 
degree  resolution. 

These  model  products  assist  fleet 
ships  in  numerous  ways,  including 
providing  information  that  helps  in 
adjusting  side  scan  sonar  and  other 


UNCLASSIFIED:  1/32°  Global  NLOM 

SSH/CURRENT  ANALYSIS:  20060705 


survey  equipment  for  specific  local 
hydrographic  conditions. 

These  products  also  help  the  ships 
to  plan  and  adjust  travel  routes  in 
anticipation  of  adverse  weather 
and  sea  conditions.  The  Navy’s 
fleet  is  not  alone  in  relying  on  the 
products  produced  on  NAVO  MSRC 
computers — the  operational  forecast 
products  and  analyses  are  pushed 
to  numerous  entities  within  the 
Department  of  Defense. 

While  this  operational  forecast  support 
is  provided  year-round,  there  have 
been  instances  where  the  METOC 
community  has  relied  particularly 
heavily  on  the  NAVO  MSRC’s 
resiliency  and  reliability.  In  2006  and 
2007  the  NAVO  MSRC  rearranged  its 
preventative  maintenance  schedule 

Continued  Next  Page... 


Temperature  (DegF) 

0  feet  2007100900  Z 

UNCLASSIFIED:  MODAS  Sea 
Temperature  at  Sea  Surface 


Operational  Products  generated  on  NAVO  MSRC  systems  by  METOC  Operational  Forecasting  Groups.  (L-R)  Navy 
Coastal  Ocean  Model  (NCOM)  36-hour  current  and  speed  forecast,  Gulf  of  Mexico;  1/32  degree  resolution  Global 
Navy  Layered  Ocean  Model  (Global  NLOM)  sea  surface  height  and  surface  current  forecast,  Hawaiian  Islands;  and 
Modular  Ocean  Data  Assimilation  System  (MODAS)  product:  sea  temperature  at  sea  surface,  Gulf  of  Mexico. 
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periods  to  accommodate  the  METOC 
group’s  oceanographic  support  of  a 
series  of  exercises  being  conducted 
by  the  Navy  fleet  in  conjunction  with 
resources  and  service  members  from 
the  Army,  Air  Force,  Marine  Corps, 
and  Coast  Guard. 

Valiant  Shield  07,  as  the  eight- 
day  Pacific  exercise  is  known, 
required  the  uninterrupted  delivery 
of  forecast  products  from  the 
NAVOCEANO  operational  team. 
These  oceanographic  products  were 
delivered  to  the  Naval  Oceanography 
Antisubmarine  Warfare  (ASW) 

Teams  (NOATs),  who  in  turn 
used  these  products  to  provide 
recommendations  on  unit  and  ASW 
sensor  deployments. 

To  assist  in  this  effort  the  NAVO 
MSRC  put  its  team  on  high  alert 
throughout  the  exercise  for  swift 
response  should  any  problems  have 
arisen  with  the  HPC  systems.  The 
overall  teamwork  between  the  MSRC 
and  the  local  METOC  operational 
community  during  such  exercises 
helps  condition  both  teams  for  any 


possible  emergency  military  exercises 
as  well. 

The  Navy  METOC  community  and 
the  Navy’s  survey  fleet  are  sometimes 
called  upon  to  provide  ocean 
forecasting,  sonar  sea-floor  mapping, 
and  pinger  location  assistance  in  non¬ 
military  recovery  and  salvaging  efforts. 
A  previous  effort  in  which  operational 
forecasts  were  used  in  the  December 
2004  Indonesian  tsunami  rescue  and 
recovery  efforts  was  detailed  in  the 
Spring  2005  issue  of  the  Navigator. 
Most  recently  the  Indonesian 
government  requested  the  Navy’s 
assistance  in  locating  the  wreckage  of 
Adam  Air  Flight  KI  574  which  went 
missing  off  the  coast  of  West  Sulawesi, 
Indonesia,  on  January  1,  2007. 

The  NAVOCEANO  survey  ship  USNS 
Mary  Sears  traveled  to  the  area  and 
on  January  9  joined  the  search  effort 
team  comprised  of  the  National 
Transportation  Safety  Board  (NTSB) 
and  the  Indonesian  Search  and 
Rescue  Command  Center  (SAR). 


The  exact  location  of  the  lost  plane 
was  unknown.  When  a  piece  of 
wreckage  washed  up  on  the  beach 
January  10,  the  NAVOCEANO  ocean 
forecasting  team  was  asked  to  project 
from  where  it  may  have  come.  Using 
surface  currents  from  the  NCOM 
model  run  daily  on  MSRC,  a  10-day 
hindcast  suggested  where  the  plane 
went  down.  By  using  a  towed  pinger 
locator  unit,  the  crew  aboard  the  Mary 
Sears  was  able  to  locate  the  downed 
aircraft’s  pinger  signal  very  near  where 
NCOM  said  it  would  be. 

This  finding  along  with  operational 
hydrographic  forecasts  and  side-scan 
sonar  charting  of  nearly  three  nautical 
miles  of  sea  floor  resulted  in  the  Mary 
Sears  crew  locating  the  wreckage  of 
the  Boeing  737  and  the  flight  data 
and  cockpit  voice  recorders  within 
several  days  of  arriving  on  scene. 

As  always,  the  NAVO  MSRC 
is  committed  to  supporting  the 
warfighter — both  in  support  of  the 
DoD  HPCMP  community  and  the 
day-to-day  operations  of  the  nation’s 
military  forces. 


NAVOCEANO  has  technical  control  of  six  329-foot-long,  5,000-ton  T-AGS  60  class  ships  and  one  hydrographic 
ship  designed  to  provide  multipurpose  oceanographic  capabilities  in  coastal  and  deep-ocean  areas.  These 
include  physical,  chemical,  and  biological  oceanography;  multidiscipline  environmental  investigations;  ocean 
engineering  and  marine  acoustics;  marine  geology  and  geophysics;  and  bathymetric  surveying. 
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Advance  Reservation  Service  Available 
on  KRAKEN 


Christine  Cuicchi,  Computational  Science  and  Applications  Lead,  NAVO  MSRC 

must  be  used  to  submit  batch  jobs  in 
the  following  manner: 


As  part  of  a  High  Performance 
Computing  Modernization  Program 
(HPCMP)  directive  to  provide 
interactive  and  real-time  High 
Performance  Computing  (HPC) 
system  access  to  the  Department  of 
Defense  (DoD)  user  base  on  an 
as-needed  basis,  the  NAVO 
MSRC  will  implement  advanced 
reservations  on  the  P4+  system, 
KRAKEN.  This  capability,  which 
was  available  previously  on  the 
Army  Research  Laboratory’s  (ARL) 
Intel  Xeon  cluster  POWELL,  will 
also  be  offered  on  the  ARL  Linux 
Networx  system,  JVN. 

This  advance  reservations  system 
will  allow  users  to  request  a  specific 
run  time  in  which  to  run  interactive 
jobs,  real-time  simulations,  or  batch 
jobs.  The  reservation  system  being 


developed  at  press  time  will  allow 
the  user  to  authenticate  on  a  front- 
end  website  via  kerberos.  First  time 
users  will  be  directed  to  request  an 
account  on  the  online  reservation 
portal.  After  establishing  an  online 
reservation  portal  account,  a  user 
may  request  a  number  of  whole 
nodes  for  a  user-specified  amount 
of  time  delineated  in  one-hour 
increments.  In  addition,  users  will 
be  able  to  specify  which  project  they 
would  like  to  use. 

When  a  reservation  request  is 
made,  the  user  will  receive  an  email 
containing  the  Load  Sharing  Facility 
(LSF)  reservation  identification 
number  for  their  job,  the  nodes  to 
which  they  have  access,  and  the 
time  of  the  reservation.  This 
reservation  identification  number 


bsub  -U  reservation  id  . <  script 

Users  who  wish  to  run  interactive 
LSF  jobs  should  submit  them  at 
the  beginning  of  the  reservation. 
Running  the  bsub  command 
will  allows  users  to  see  a  list  of 
reservations  as  well. 

All  reservations  will  begin  at  the 
time  the  user  requests.  Users  will 
have  the  ability  to  cancel  their 
reservations  up  to  30  minutes  before 
the  reservation  start  time.  If  the 
reservation  is  not  cancelled,  system 
utilization  will  be  charged  for  these 
nodes  regardless  of  how,  or  if,  the 
nodes  are  used. 

The  initial  deployment  will  allow 
for  reservations  on  256  processors 
(32  nodes)  and  will  likely  increase 
to  512  processors  once  the  stability 
of  the  process  has  been  established. 
The  NAVO  MSRC  will  continually 
monitor  the  need  for  this  advance 
reservation  capability  and  make 
adjustments  as  necessary. 

Implementation  and  monitoring  of 
the  advance  reservations  system  is 
courtesy  of  team  members  Grant 
Black,  Patrick  Thompson,  and  Lee 
Whatley,  all  of  Lockheed  Martin 
Mission  Services  (LMMS).  For  more 
information  on  how  to  schedule  an 
advance  reservation,  please  visit  the 
NAVO  MSRC  website  at  http://www. 
navo.hpc.mil. 

Screen  shot  of  the  NAVO  Advance 

Reservation  System. 
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STEVEN  W.  BUNTE,  MARGARET  M.  HURLEY,  DECARLOS  E.  TAYLOR 

u.s.  army  Research  Laboratory 
ABERDEEN  PROVING  GROUND,  MD  21005 


Daniel  f.  Curran 
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ENSCO,  INC. 

MELBOURNE,  FL  32940 


ISTRY  MODELING  OF 
RIC  FATE  OF 

jowpou ms  (TICS) 


Predicting  the  atmospheric  fate  of  Toxic  Industrial  Compounds 
(TICs)  is  a  critical  component  of  the  Department  of  Defense  (DoD) 
chemical/biological  defense  programs  as  well  as  national  security. 
Typically,  Transport  and  Dispersion  (T&D)  models  (e.g..  Second- 
order  Closure  Integrated  PUFF  Model  (SCIPUFF)  or  the  recently 
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Figure  1.  Plume  model  showing  the  contamination  zone  at  9am  following  the  release  of  2-methylpropane  at  8am. 
Figure  2.  Plume  model  showing  the  contamination  zone  at  noon  following  the  release  of  2-methylpropane  at  8am. 
Figure  3.  Plume  model  showing  the  contamination  zone  at  3pm  following  the  release  of  2-methylpropane  at  8am. 
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This  innovation  has  resulted  in  an 
improved  ability  to  model  the  fate 
of  TICs  in  the  atmosphere,  resulting 
in  more  realistic  representations  of 
contamination  zones.  Such  T&D 
methods  can  be  used  not  only  to 
predict  the  changing  health  threat 
in  both  urban  and  battlefield 
environments,  but  can  also  predict  the 
nature  of  the  resulting  by-products. 
This  information,  in  turn,  would  be 
valuable  in  programming  advanced 
sensors  to  warn  of  the  release  of  a 
toxic  compound  even  if  the  material 
has  degraded  by  the  time  it  reached 
the  detection  array.  In  addition,  the 
results  of  these  calculations  can  also 
be  used  to  make  a  more  scientifically 
defensible  selection  of  simulants  for 
challenging  detectors,  as  well 
as  assisting  in  the  evaluation  of 
both  individual  and  collective 
protection  systems. 

These  calculations  could  also  be 
used  to  provide  a  sound  theoretical 
underpinning  to  the  development 
of  physical  properties  data,  which 
is  critical  for  choosing  the  proper 
simulant  for  use  in  synthesis,  fate,  and 
decontamination  studies,  as  well  as  in 
the  assessment  of  new  potential  threat 
agents  and  the  selection  of  improved 
decontamination  concepts. 


Figures  1,  2,  and  3  each  demonstrate 
the  importance  of  including  finite 
chemistry  in  Transport  and  Dispersion 
(T&D)  models.  Each  figure  is  a 
snapshot  in  time  showing  the 
plume  following  the  release  of  2- 
methylpropane  at  8  a.m. 

The  simulation  incorporates  realistic 
meteorological  data  and  assumes  a 
uniform  release  of  the  material.  The 
figures  each  include  two  plots:  one 
with  the  chemistry  turned  “off”  and 
the  other  with  chemistry  turned  “on.” 
The  color  contours  within  each  plume 
are  computed  concentrations  of  the 
parent  compound. 

In  Figure  1,  both  plumes  are  similar 
in  shape,  aerial  coverage,  and 
concentration.  Three  hours  later,  as 
shown  in  Figure  2,  the  shape  of  the 
plumes,  the  aerial  coverage,  and  the 
concentrations  of  2-methylpropane 
are  very  different.  This  is  primarily 
due  to  an  increase  in  the  sun  angle, 
which  results  in  a  higher  concentration 
of  hydroxyl  (OH)  in  the  atmosphere. 
Finally,  Figure  3  shows  the  plume  6 
hours  after  release  and  3  hours  later 
than  the  plume  shown  in 
Figure  2. 


The  aerial  coverage  of  the  two  plumes 
in  Figure  3  is  radically  different.  This 
series  of  snapshots  underscore  the 
importance  of  including  accurate 
chemistry  into  the  T&D  models  so 
that  a  realistic  representation  of  the 
contamination  zone  can  be  predicted. 

Accurate  rate  constants  can  be 
calculated  using  a  combination 
Quantum  Chemistry  (QC)/Chemical 
Dynamics  (CD)  approach.  The  QC 
method  is  computationally  intensive 
as  it  requires  the  use  of  high  level  ab 
initio  quantum  chemistry  to  calculate 
accurate  structures  of  the  TICs  at  the 
various  important  stationary  points 
along  the  minimum  energy  path  that 
connects  reactants  to  products. 

For  the  QC  work,  we  use  the  ACESIII 
quantum  chemistry  package.  ACESIII 
is  the  parallel  version  (developed 
under  Common  High  Performance 
Computing  Software  Initiative 
(CHSSI)  funding,  project  CBD-03)  of 
the  ACESII  program,  which  itself  was 
developed  over  the  last  20-plus  years 
under  support  from  the  Air  Force 
Office  of  Scientific  Research  (AFOSR), 
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Office  of  Naval  Research  (ONR),  and 
Army  Research  Office  (ARO)T 

The  ACESIII  is  a  general  QC  package 
focusing  on  many  body  methods  (i.e., 
finite-order  Many  Body  Perturbation 
Theory  (MBPT)  and  infinite  order 
Coupled  Cluster  (CC)  methods)  and  is 
used  to  compute  molecular  structures, 
energies,  vibrational  spectra,  electronic 
spectra,  magnetic  properties, 
polarizabilities,  etc. 

The  ACES  program  package  was 
developed  by  Professor  Rodney 
Bartlett  and  co-workers  at  the 
Quantum  Theory  Project  of  the 
University  of  Florida.  The  QC 
information  is  then  input  into  a 
recently  developed  Semi-Classical 
Flux-Flux  Autocorrelation  Function 
(SCFFAF)  chemical  dynamics  code, 
which  computes  the  k(T)  of  the 
reaction  under  investigation  over  a 
wide  temperature  range.  Details  of 
this  methodology  can  be  found  in 
K.  Runge  and  M.  G.  Cory  and  R.  J. 
Bartlett,  “The  Calculation  of  Thermal 
Rate  Constants  for  Gas  Phase 
Reactions:  A  Quasi-Classical  Flux-Flux 
Autocorrelation  function.  ”2 

The  Reaction  of  Dimethyl 
Phosphonate  (DMHP)  + 
Hydroxyl  Radical  (OH) 

When  released  into  the  troposphere, 
Dimethyl  Phosphonate  (DMHP)  will 
degrade,  for  example,  by  reacting 
with  OH  radicals.  The  products  of 
this  hydrogen  abstraction  reaction  are 
a  radical  DMHP  species  and  water. 

Our  calculations  found  four  transition 
states  that  are  accessible  at  ambient 


temperatures  and  are  thus  the  major 
contributors  to  the  total  rate. 

Figure  4  shows  a  representative 
transition  state  for  the  abstraction 
of  a  hydrogen  atom  from  one  of 
the  methyl  groups  on  DMHP;  this 
transition  state  is  also  the  major 
contributor  to  the  total  rate.  Figure 
5  shows  the  computed  temperature 
dependent  rate  constants  for  each  of 
these  mechanisms,  as  well  as  the  total 
rate  constant,  which  is  simply  the  sum 
of  the  individual  rate  constants.  Our 
calculated  total  rate  constant  agrees 
quite  favorably  with  the  experimental 
rates  measured  by  Atkinson. 3  Note 
that  each  of  the  experimental  data 
points  is  an  individual  measurement 
at  a  single  temperature,  whereas  our 
calculations  give  the  full  temperature 
dependent  rate. 


Conclusions 

In  this  work,  we  have  used  a 
combination  of  quantum  chemistry 
and  chemical  dynamics  to 
theoretically  predict  the  temperature 
dependent  rates  for  the  abstraction 
of  hydrogen  atoms  by  OH  radicals 
from  dimethyl  phosphonate.  These 
computed  rate  constants  will  now 
be  used  as  input  into  atmospheric 
chemistry  modules  within  DoD 
T&D  models,  such  as  SCIPUFF 
and  ChemCODE/ChemCONC.  The 
addition  of  finite  chemistry  will  result 
in  the  improved  ability  of  the  models 
to  handle  the  fate  and  interaction  of 
TICs  in  the  atmosphere,  resulting  in 
a  more  realistic  representation  of  the 
contamination  zone. 


250  260  270  280  290  300  310  320  330  340  350 


Temperature 

Figure  5.  Temperature  Dependent  Rate  Constants  for  the  Reaction  of 
DMHP  +  OH. 
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Setting  Standards  in  the  Technology 
Insertion  Process 

Christine  Cuicchi,  Computational  Science  and  Applications  Lead,  NAVO  MSRC 


The  one  constant  in  the  world  of 
High  Performance  Computing  (HPC) 
is  change.  As  computing  technology 
grows  ever  more  sophisticated,  greater 
computing  power  becomes  available. 
To  keep  this  power  harnessed  for 
use  by  Department  of  Defense 
(DoD)  users,  the  High  Performance 
Computing  Modernization  Program 
(HPCMP)  administers  the  annual 
Technical  Insertion  (TI)  process. 

The  TI  process  encompasses  staff 
members  of  the  four  HPC  Major 
Shared  Resource  Centers  (MSRCs), 
the  Shared  Resource  Centers  (SRCs), 
and  advisors  from  selected  support 
contract  companies.  These  individuals 
are  selected  to  tap  into  their  HPC  and 
User  Support  expertise  to  ensure  that 
the  DoD’s  annual  procurement  of 
HPC  systems  ($40-60  million  worth 
in  TI-08)  is  used  to  provide  the  best 
environment  and  technology  for  users 
now  and  in  the  future. 

To  streamline  efforts,  participants  are 
divided  into  three  teams:  Usability, 
Performance  (which  evaluate  vendor 
proposals),  and  Document  preparation 
(which  prepares  TI  Request  for  Quotes 
(RFQs)). 

Since  2000,  NAVO  MSRC  staff  and 
HPC  resources  have  been  intimately 
involved  in  all  aspects  of  the  intricate 
HPCMP  TI  process. 

Benchmarking 

As  the  TI  process  has  matured  over 
the  past  seven  years,  the  Performance 
Team  has  created  and  enhanced  a 
TI  application  benchmark  suite,  a 
synthetic  benchmark  suite,  and  a 
performance  prediction  system.  As 
a  vendor  submits  an  application, 
the  Team  evaluates  a  vendor’s 


performance  against  these  pre- 
established  benchmarks  based  on 
the  vendor’s  ability  to  compile  their 
application  and  then  runs  it  against 
the  benchmarks  within  a  specified 
speedup  over  the  standard  DoD 
system’s  TI  application  run  times. 

The  TI-08  application  suite  consists 
of  applications  chosen  to  represent 
the  overall  requirements  of  the  DoD 
HPCMP  user  base: 

•  AERO 

•  COBALT 

•  CTH 

•  GAMESS 

•  HYCOM 

•  OOCORE 

•  AMR 

•  ICEPIC 

•  LAMMPS 

•  OVERFLOW-2 

•  WRF 

Vendors  are  also  evaluated  on 
system  performance  metrics  gathered 
via  both  the  synthetic  benchmarks 
and  the  performance  prediction 
system  developed  at  the  San  Diego 
Supercomputing  Center. 

NAVO  MSRC  staff  member 
Christine  Cuicchi  and  Productivity 
Enhancement  Technology  Transfer 
(PET)  on-site  Dr.  John  Cazes  (Texas 
Advanced  Computing  Center  (TACC)) 
have  served  on  the  Performance  Team 
since  the  TI-02  and  TI-05  efforts, 
respectively.  For  the  TI-08  effort,  Dr. 
Cazes  remains  on  this  team  while  Ms. 
Cuicchi  has  joined  the  Usability  Team. 

The  Usability  Team  reviews  vendor 
proposals  for  robustness  and  ease  of 
use  based  on  a  number  of  criteria, 
including  the  availability  of  key 


commercial  software  packages, 
system  utilities,  and  system 
characteristics  that  facilitate 
integration,  operation,  maintenance, 
and  upgrades.  Past  Usability  Team 
members  from  the  NAVO  MSRC 
have  included  Dr.  Cazes  (TACC), 

Ed  Farrar  (Lockheed  Martin  Space 
Operations  (LMSO)),  Lee  Whatley 
(Lockheed  Martin  Mississippi  Space 
&  Technology  Center  (LMMS)),  and 
Dave  Cole  (NAVO  MSRC),  who  led 
the  team  for  the  TI-06  process.  Mr. 
Whatley  is  once  again  serving  on  the 
Usability  Team  for  the  current  TI- 
08  process,  and  Mr.  Cole  has  been 
serving  as  the  Document  Preparation 
Team  lead  since  TI-07. 

NAVO  MSRC  TI  Benchmarking 

NAVO  MSRC  HPC  systems  have  also 
been  an  important  part  of  the  TI-XX 
process  over  the  past  several  years. 
Each  year,  one  production  system 
in  the  HPCMP  is  selected  to  be  the 
DoD  standard  performance  system, 
against  which  vendor  proposals  and 
performances  will  be  evaluated. 

In  TI-03,  the  NAVO  MSRC  1024 
processor  IBM  Power3  system  HABU 
was  chosen  to  be  the  DoD  standard 
system.  HABU  served  as  the  standard 
system  until  TI-05,  when  it  was 
replaced  by  the  NAVO  MSRC  TI-04 
acquisition  system  KRAKEN  -  an  IBM 
Power4+  2944  processor  machine. 

In  2007,  the  Engineer  Research  and 
Development  Center  (ERDC)  MSRC 
Cray  XT4,  SAPPHIRE,  was  selected 
by  the  Performance  Team  to  be  the 
TI-08  standard  system,  thus  ending 
the  reign  of  NAVO  MSRC  systems  as 
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representative  of  the  DoDs  required 
performance  ability.  However,  HABU 
remains  the  HPCMP’s  best  example 
of  a  balanced  system — production 
systems  and  proposed  systems  are  still 
compared  with  the  performance  of  the 
1024  processor  HABU. 

Another  important  component  of 
the  NAVO  MSRC’s  support  of  the 
TI  process  has  been  support  of  the 
benchmarking  efforts,  both  to  garner 
standard  system  runtimes  as  well 
as  benchmarking  the  performance 
of  other  production  HPC  systems 


installed  at  the  NAVO  MSRC.  In  the 
initial  years  of  the  TI  process,  the  staff 
at  each  MSRC  was  responsible  for 
running  the  entire  applications  suite 
on  each  production  HPC  system  at 
their  site.  This  task  has  since  fallen 
under  the  auspices  of  the  ERDC 
MSRC’s  Computational  Science 
and  Applications  (CS&E)  Team 
and  the  Performance  Modeling  and 
Characterization  (PMaC)  laboratory 
at  the  San  Diego  Supercomputing 
Center  (SDSC).  NAVO  MSRC  staff 
continues  to  work  closely  with  these 


teams  to  ensure  fast  turnaround  and 
proper  support  to  keep  within  the 
tight  TI  schedule. 

Conclusion 

The  NAVO  MSRC  is  proud  to 
have  been  a  substantial  part  of  the 
Technology  Insertion  process  and 
looks  forward  to  continuing  to  provide 
technical  expertise  and  HPC  system 
support  in  assisting  the  HPCMP 
in  selecting  the  most  effective  and 
efficient  HPC  systems  for  its  users. 


Three  members  of  the  NAVO  MSRC  TI  08  team:  (L-R)  Lee  Whatley  (Lockheed  Martin  Mississippi  Space  & 
Technology  Center  (LMMS),  Christine  Cuicchi  (Computational  Science  and  Applications  Lead,  NAVO  MSRC),  and 
Dr.  John  Cazes  (Texas  Advanced  Computing  Center  (TACC)). 
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Navigator  Tools  and  Tips 

LSF  Top  5  FAQs 

Sheila  Carbonette,  NAVO  MSRC  User  Support 

The  Computing  Load  Sharing  Facility  (LSF)  by  Platform 
Computing  is  the  batch  scheduler  currently  used  on 
the  NAVO  MSRC  IBM  systems  KRAKEN,  PASCAL,  and 
ROMULUS.  This  issue  of  Tips  ‘n  Tricks  is  designed  to 
provide  answers  to  some  of  our  user’s  most  frequent 
questions. 

Q.  Where  can  I  learn  how  to  use  LSF? 

A.  An  introductory  guide  can  be  found  on  the  NAVO 
MSRC  website.  The  URL  of  the  LSF  introduction  is: 
http://www.  navo.  hpc .  mil/lsf_guide .  html 

The  guide  provides  an  overview  of  LSF,  a  comparison  of 
LoadLeveler  and  LSF  commands  and  example  scripts. 

Q.  How  do  I  make  one  batch  job  wait  for  another  to 
complete? 

A.  LSF  handles  this  through  the  use  of  job  dependencies 
with  the  "bsub"  command  wait  option,  "-w".  The  wait  op¬ 
tion  allows  you  to  specify  the  condition  which  you  wish  to 
wait  for  before  starting  the  job. 

An  example  batch  script  that  uses  the  wait  option  to  start 
the  second  job  at  the  completion  of  the  first  job  follows: 

kraken%  cat  jobOl 
#BSUB  -P  N AVO  SLM A 
# B  S  U  B  -o  %J  .log 
#BSUB  -e  %J  .err 
# B  S  U  B  -J  jobOl 

#  B  S  U  B  -N 

#  B  S  U  B  -W  1:30 

# B S  U  B  -R  "  spa n [ pti I e=  8 ] " 

# B  S  U  B  -n  8 
# B  S  U  B  -q  standard 

# 

#  RUN  PARALLEL  EXECUTABLE 
./mpirun.lsf  mpijob.exe 

# 

#  SUBM IT  SECOND  J  OB 
bsub  <  j o b 0 2 

# 

#  END  OF  jobOl  SCRIPT 


kraken%  cat  job02 

#  B  S  U  B  -P  NAVO  SLM  A 
#BSUB  -o  %J  .log 

#  B  S  U  B  -e  %J  .err 

#  B  S  U  B  -J  j  o  b 0  2 

# B  S  U  B  -w  'done("  jobOl" )' 

# B  S  U  B  -N 

#  B  S  U  B  -W  0:30 

#  B  S  U  B  -R  "  spa  n  [  pti  I  e=  8] " 

#  B  S  U  B  -  n  8 

#  B  S  U  B  -q  standard 

# 

#  RUN  PARALLEL  EXECUTABLE 
./mpirun.lsf  mpijob2.exe 

# 

#  END  OF  job02  SCRIPT 

Additional  information  on  the  wait  option  can  be  found 
in  the  bsub  main  page. 

Q.  My  parallel  batch  job  failed  with  the  following  mes¬ 
sage: 

"Cannot  find  enough  ntbl  windows  on  sniO. 

E  xiting  ..." 

What  does  this  error  mean? 

A.  This  error  can  mean  a  couple  of  things.  The  easiest  to 
check  first  is  the  quota  on  your  home  directory.  If  you  have 
exceeded  your  quota,  or  are  near  exceeding  your  quota, 
then  LSF  can't  write  required  host  files  to  your  home  direc¬ 
tory.  The  following  is  an  example  listing  of  the  file  names: 

kraken%  Is  -al  .sni*  .windows*  .all. hosts.*  .host, 
list.* 

-rw-rw -  1  shecar  NAV0SLMA  10483  Sep 

23  13:57  .sniO. 289394 

-rw-rw -  1  shecar  NAV0SLMA  10483  Sep 

23  13:57  .sniO. 289394.1 

-rw-rw -  1  shecar  NAVOSLMA  10563  Sep 

23  13:57  .snil. 289394 

Continued  Page  22 
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Tom  Dunn,  NAVO  MSRC  Director;  CAPT  Kusters,  Prospective  Commam 
cer  of  FNMOC;  Dave  Cole,  Christine  Cuicchi,  NAVO  MSRur 


James  Rigney,  NAVOCEANO;  Dr.  Ming  Ji,  Director  of  NOAA’s  Na 
tional  Environmental  Prediction  Center’s  Ocean  Prediction 
Center;  Pete  Gruzinskas,  NAVO  MSRC;  Joseph  Sienkeiwicz  and 
Robert  Daniels,  NCEP  OPC. 


John  Choplinsky,  Deputy  Comptroller,  U.S.  Fleet  Forces 
Command 
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Basic  Oceanography  Accession  Training  (BOAT)  class  tour  the  Tom  Dunn,  NAVO  MSRC  Director  and  John  Dupuis,  NAVO  MSRC, 


MSRC  with  Christine  Cuicchi,  NAVO  MSRC. 


brief  visitors  from  Information  Management  Resources,  Incorpo¬ 


rated  (IMRI) 


CAPTRobert  E.  Kiser,  CNMOC;  CAPT  Brian  B.  Brown,  NAVOCEANO;  CAPT  David  Titley,  CNMOC;  CAPT  John  D.  Cousins, 
NAWJCEANO;  RDML  Timothy  McGee,  CNMOC;  RADM  David  Gove,  Oceanographer  of  the  Navy;  Tom  Dunn,  Director  NAVO 
MSRC;  Ed  Gough,  Deputy/Technical  Director  CNMOC;  John  Lever,  CNMOC;  CAPT  Jeffrey  Best,  OPNAV. 


Dave  Cole,  NAVO  MSRC;  To?n  Dunn,  NAVO  MSRC  Director;  and  Ed  Gough,  Deputy/Technical  Director 
CNMOC  brief  Scott  McNealy,  Chairman  of  Sun  Micf&syStems  (center)  and  party  at  the  NAVO 
Major  Shared  Resource  Center 


Industry  Education  Partnerships  Workshop  sponsored  by  Mississippi  Stati 
National  Science  Foundation 


i 

mm 

-rw-rw -  1  shecar  NAVOSLMA 

23  13:57  .sni 1.289394.1 

-rw-rw -  1  shecar  NAVOSLMA 

23  13:57  .windows. 289394.1 

-rw-rw -  1  shecar  NAVOSLMA 

13:57  .all. hosts. 289394 

-rw-rw -  1  shecar  NAVOSLMA 

23  13:57  .host. list. 289394 


10563  Sep 
10214  Sep 
682  Sep  23 
5456  Sep 


You  can  check  your  quota  with  the  /site/bin/quota 
command. 

kraken%  /site/bin/quota 

Block  L imits 

F ilesystem  type  KB  quota  limit  in 
doubt  grace 

gpfs_  hm  USR  517364  512000  524288 

2054  expired 

(Note:  The  output  from  the  quota  command  has  another 
section  for  “File  Limits”  that  was  omitted  in  this  example.) 

This  second  reason  can  be  that  your  job  is  not  getting  all  of 
the  adapter  windows  available  on  a  given  node.  To  ensure 
your  job  does  not  start  until  all  of  the  adapter  windows  are 
available  on  a  node,  the  following  resource  requirement 
can  be  added  to  your  batch  script: 

For  KRAKEN: 

#BSUB  -R  ' rusage[ntbl  windows=  16]span[ptile=8]' 

For  BABBAGE: 

#BSUB  -R  ' rusage[ntbl  windows=32]span[ptile=  16]' 

Another  useful  command  to  get  this  information  as  well  as 
your  allocation  usage  is  the  "show_usage"  command. 

For  example: 

/kraken%  /site/bin/show_usage 


The  easiest  way  to  determine  the  project  name,  is  to 
type  the  unix  command,  groups.  At  the  MSRC,  the 
project  identifier  is  the  same  as  the  group  name.  For 
example: 

kraken%  groups  shecar 
shecar:  N  RLSS018 

Another  useful  command  to  get  this  information  as  well  as 
your  current  allocation  is  the  show_  usage  command.  For 
example: 


kraken%  /site/bin/show_usage 

Date  Time 

System  Account 

10/01/07  12:16:13 

kraken  N  R  LSS008 

Allocated 

20000.00 

U  sed 

0.00 

B  alance 

20000.00 

Percent 

0.00  used 

Q Where  can  I  go  to  get  more  help? 


A.  The  questions  addressed  in  this  article  and  others  are 
found  in  the  Frequently  Asked  Questions  (FAQ)  web  page 
available  at  the  following  URL:  http://www.navo.hpc.mil/ 
user_tips.html 

This  FAQ  page  also  includes  information  and  links  to 
help  you  get  oriented  with  the  different  High  Performance 
Computing  (HPC)  systems  and  the  Archive  servers.  If  you 
can't  find  an  answer  to  your  question  on  this  page,  you  can 
always  call  the  Consolidated  Customer  Assistance  Center 
(CCAC)  or  NAVO  MSRC  User  Support  for  assistance: 

The  Consolidated  Customer  Assistance  Center  (CCAC)  is 
available  0700-2200  (Central  Time),  Monday-Friday  for 
issues  of  an  unclassified  nature: 


Q.  When  I  try  to  submit  a  job,  I  get  the  error  message: 

"shecar  is  not  authorized  to  use  project 
N  R  LSS03755018  . 

How  do  I  find  out  my  project  name? 

A.  The  NAVO  MSRC  uses  an  eight-character  naming 
convention  for  projects  instead  of  the  13  character  full 
sub-project  identifier.  The  NAVO  MSRC  eight-character 
identifier  is  composed  of  the  five-character  Organization  ID 
and  the  three-character  allocation  number  assigned  by  the 
Service/Agency  Approval  Authority  (S/AAA). 

For  example:  Full  Subproject  Identifier:  NRLSS03755018 
NAVO  MSRC  Project/Group  Identifier:  NRLSS018 


•  Help  Desk  toll  free  number:  1-877-CCAC-039  or 
1-877-222-2039 

•  Help  Desk  email  address:  help@ccac.hpc.mil 

•  Website:  http://www.ccac.hpc.mil 

NAVO  MSRC  User  Support  is  available  0800-1630 
(Central  Time),  Monday-Friday,  for  issues  of  a  classified 
nature: 

•  User  Support  toll  free  number:  1-800-993-7677 

•  Help  Desk  E-mail:  msrchelp@navo.hpc.mil 

•  Website:  http://www.navo.hpc.mil 
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